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Introduction 

The National Board for Professional Teaching Standards' system of advanced, 
voluntary certification for k-12 teachers represents the state-of-the-art in complex 
performance assessment of teaching. The six site-based portfolio exercises and the four 
on-demand assessment center exercises constitute the most comprehensive assessment of 
teaching yet devised. Although the assessment entails exercises that contain student 
work which the candidate must discuss and analyze, student achievement and the quality 
of student products, per se, are not a part of the assessment. Moreover, although 
candidate teachers must submit two video tapes (and associated commentary) which 
feature their instructional expertise in whole class and small group settings, the NBPTS 
assessment does not entail actual classroom visitation and observation by trained 
observers. 

The present investigation is intended to provide relevant information on these two 
aspects of the NBPTS assessment system. More specifically, the present study will 
attempt to provide answers to the following two questions: 

1 . Do students who have been taught by a National Board Certified teacher 
produce higher quality work than students of teachers who attempted 
certification but were not certified? 

2. On the basis of observation protocols that were developed separately and apart 
from the National Board, are the observed classroom practices of National 
Board certified teachers demonstrably different from those of teachers who 
attempted certification but were not certified? 
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A third question addressed in the study involves a comparative examination of the 
extent and type of professional activities and involvement of Board certified teachers 
versus teachers who were not certified. More specifically, 

3. Do National Board certified teachers and teachers who were not certified differ 
in the number, variety, and nature of their professional activities and 
involvement? 

This report will provide a brief summary of the data collection activities, and ' . 
plans for scoring and analyzing the data. 



Comparative Teaching Practices and Outcomes 
Fourteen dimensions of expert teaching 

Through a comprehensive review of the literature on expertise in teaching as well 
as other domains, we have identified the following 14 dimensions of expert teaching that 
will be used to answer the first two questions in the previous section: 

1 . Content knowledge. The expert has a deep understanding of the content to be 
taught to students. 

2. Pedagogical knowledge. The expert transforms essential aspects of subject matter 
to connect with students’ ways of understanding. These transformations include 
the following: 

2A Identifying essential representations 

2A1 Expert teachers have deeper representations about teaching 
and learning. 

2A2 Expert teachers are problem solvers. 

2B Setting goals for diverse learners 

2B1 Expert teachers can anticipate, plan, and improvise. 

2C Guiding learning through classroom interactions 

2C1 Expert teachers have excellent proficiency for creating an 
optimal classroom climate for learning. 

2C2 Expert teachers have a multidimensional perception of class 
situations. 
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2C3 Expert teachers are more context-dependent and have high 
situation cognition. 

2D Monitoring learning and providing feedback 

2D1 Expert teachers are more adept at monitoring and provide 
much feedback to students. 

2D2 Expert teachers are more able to check and test-out their 
hypotheses or strategies. 

3. Affective attributes. The expert teacher has the following affective attributes: 

3 A High respect for students 

3B Much passion about teaching and learning 

4. Learning outcomes. The expert teacher promotes the following: 

4A Motivation & Self-efficacy 
4C Challenge of objectives 

4D Outcomes of lessons: deep understanding and achievement 

The teacher sample 

The sample included 65 teachers from throughout North Carolina, Ohio, and the 
Washington, DC area who have sought National Board certification as either Middle 
Childhood Generalists (MCGen) or Early Adolescence/English Language Arts teachers 
(EA/ELA). The sample consisted of teachers whose final assessment score falls within 
four score groups that have been identified for comparison, as follows: 



Score Group 


Candidates whose total score on the assessment is: 


1 


at least one and one-fourth standard deviations below the passing 
score 


2 


between one-fourth, and three-fourths of a standard deviation 
below the passing score 


3 


between one-fourth, and three-fourths of a standard deviation 
above the passing score 


4 


at least one and one-fourth standard deviations above the passing 
score 
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The following table shows the demographic composition of the teacher sample. 

EA/ELA MC/Gen 

White African White African 
American American 

7 7 3 4 3 

s' 

J 2 10 1 5 1 

2 ? 

© 3 5 8 2 

£ 

4 7 1 7 1 

Data collection procedures 

The primary data collection activity for this portion of the study involved 
observing classroom instruction related to a unit of instruction that had previously been . 
described by the teacher. Prior to the classroom visit, the teacher completed an open- 
ended questionnaire in which she described how she planned the lesson to be observed, 
and also discussed her teaching philosophy. 

A pair of trained observers, blind as to the teacher’s certification status, visited the 
classroom and observed one to three hours of instruction. Observer One coded the 
Narrative Record, an open-ended form on which the observer described classroom activity 
and teacher-student interactions. Observer Two coded the Observation Protocol, a 
structured observation form on which the observer coded the amount and nature of 
feedback, management strategies, and off-task behaviors. Immediately following the 
lesson, the observers conducted structured interviews of three randomly-selected students 
from the class that they had observed. Following the student interviews, the observers 
conducted a structured interview of the teacher. The lesson, student interviews, and 
teacher interview were tape recorded and later transcribed. 
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At the conclusion of the visit, the observers each completed semantic differential 
scales consisting of Likert-scale items designed to measure the degree of passion and 
multidimensionality exhibited by the teacher. 

The observers left a packet of teacher and student questionnaires and grade- 
appropriate writing prompts to be administered and returned at a later date. Along with 
the packet of completed questionnaires and writing samples, the teacher submitted a log of 
all assignments made during the instructional unit and all work submitted by four 
randomly-selected students in response to those assignments. 

The Pre-observation Questionnaire, Narrative Record, Observation Protocol, and 
transcriptions of the lesson and interviews were stripped of potentially identifying 
information and compiled into a casebook for scoring. 

Comparative Professional Activities and Agencies’ Use of NBCTs 

In order to investigate the extent to which National Board Certified teachers differ 
from non-certified teachers in the amount and type of professional activity, we designed 
and administered an extensive telephone interview protocol to a sample of 40 MC/Gen 
and EA/ELA candidates from across the United States. 

We also explored the ways in which various agencies are making use of the 
expertise of National Board Certified teachers by interviewing a sample of 40 NBCTs 
from all certificate areas across the United States. 

Analytical Procedures 

The focus of the analytical procedures we intend to employ centers on 
distinguishing between Board certified teachers and non-certified teachers. Several 




Page 10 



11 



Preliminary Analysis Report 



multivariate procedures will be employed. As described in more detail elsewhere, the 
database consists of a multi-component casebook for each teacher, student work and 
writing samples, and both teacher and student questionnaires. The intended analytical 
strategy contemplates separate analyses of each major data source. In addition to the 
analyses described below, summary descriptive statistics (means and standard deviations) 
for all dimensions, student work, and questionnaires will be provided for both certified and 
non-certified teachers. ' v*-* ' • > 

Analysis of Teaching Practice: Casebooks 

As noted earlier, the casebooks for each of the 65 teachers in the sample contain 
information on teaching competence from six sources: the pre-observation questionnaire, 
the narrative record, the classroom observation protocol, the lesson transcript, student 
interviews, and teacher interviews. Scorers are being trained to examine each casebook 
component systematically for evidence of each of the 14 teaching dimensions. They then 
make a holistic dichotomous (0-1) judgment in response to the following question: Taken 
in total, is the overall evidence present in this casebook regarding dimension X indicative 
of an accomplished teacher? 

The above process results in a 14 element 0-1 vector for each teacher in the 
sample. These data will be analyzed via a discriminant function analysis (DFA). 
Discriminant function analysis obtains that linear combination of the 14 dimensions that 
maximally discriminates between certified and non-certified teachers. It thus provides 
information regarding one of the essential questions of the study, "Can National Board 
Certified teachers be reliably distinguished from non-Board Certified teachers on the basis 
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of relevant information about their classroom practice?" In addition to a test of the 
statistical significance of the discrimination, DFA provides discriminant function weights 
for each dimension. DFA weights allow us to determine precisely which dimension(s) 
contribute most and which contribute least to the discrimination. It is noted in passing 
that a two-group discriminant function analysis, like the one proposed here, is formally 
identical to an ordinary multiple regression with a dichotomous dependent variable. Thus, 
in this special case, DFA weights and regression weights are identical. * ' ‘ > " ' 

Analysis of Teaching Outcomes: Student Writing and Work Samples ' • > 

The two sources of "student achievement" in this investigation were (1) writing 
samples to grade-appropriate writing prompts provided by the research team, and (2) 
work samples of four randomly-selected students per teacher produced in response to 
assignments made as part of the specific unit of instruction observed during the visitation. 
For reasons outlined below, these indices of student achievement will be analyzed 
separately. 

Before describing procedures used to examine the relationship between student 
achievement and National Board Certification, it is necessary to briefly state the rationale 
for our choice of measures of student achievement. We appreciate the intense interest in 
the question "Do students who have been instructed by National Board Certified teachers 
obtain higher scores on standardized measures of academic achievement than students 
who have not been taught by NBCTs?" Nevertheless, a simple comparison of mean scores 
on standardized achievement tests of students who have been instructed over the past 
academic year by NBCTs versus those who were not is misguided. First, such a 
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comparison ignores other powerful influences on standardized measures such as socio- 
economic status. Even if one could control for such effects, a simple mean comparison 
would still be inappropriate. It is simply unrealistic to expect student performance on off- 
the-shelf, multiple-choice achievement tests to be measurably sensitive to differences 
between teachers in a single academic year. This is especially so since the relationship 
between what teachers are teaching and the content of various standardized tests varies 
widely from state to state, district to district, school to school, and indeed from classroom 
to classroom. 

Student Writing. In the analysis of the writing sample data, we will compare 

\ l 

writing scores of students of NBCTs with those of students of non-NBCTs after 
controlling for entering ability as indexed by achievement test scores. Ideally, we would 
prefer to condition on an entering test of writing ability, but no such measure is available. 
We fully appreciate that scores on multiple-choice achievement tests are at best a 
compromise covariate. 

Student work samples. In a small preliminary study, Hattie et al. (1996) noted a 
tendency for students taught by NBCTs to differ not so much in how much they knew, but 
in the quality of their knowledge and understanding. That is, he found that students taught 
by NBCTs, when compared to those taught by non-NBCTs, tended to exhibit a greater 
depth of understanding of a given subject. In particular, students taught by NBCTs 
produced work and provided oral explanations of concepts that indicated understanding at 
a deeper level than students taught my non-NBCTs. If confirmed with larger samples, this 
is a potentially important finding. We will subject Hattie's preliminary conclusion to a 
more thorough test with the larger sample obtained in the present investigation. Biggs & 
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Collis (1982) has developed a SOLO taxonomy and associated scoring scheme that allows 
student productions to be evaluated on a 4-point scale from "1", a "surface" understanding 
of a concept to "4", a "deep structure" understanding. 

The approximately 260 student work samples produced in response to assignments 
made during the observed unit of instruction will be scored using the SOLO taxonomy. 
Unfortunately, a straightforward comparison of mean scores on the SOLO taxonomy for 
students taught by NBCTs versus those taught by non-NBCTs would be misleading? This ■ 
is so because the nature of student products depends critically upon the nature of the 
assignment given by the teacher. Some assignments (e.g., a piece of purely descriptive 
writing) do not lend themselves to a scoring analysis using the SOLO taxonomy since they 
are deliberately intended to be "surface" level exercises. With the aid of our teachers-in- 
residence, we will remove from the analysis all exercises that cannot be scored using the 
SOLO scoring scheme. Mean SOLO taxonomy scores of students taught by NBCTs and 
students taught by non-NBCT students on the remaining student products will be 
compared. 

Analysis of Student and Teacher Affect 

Student and teacher attitudes toward education and learning are important but 
often neglected aspects of schooling. As described in previous reports, we also obtained 
attitudinal and other affective information via students and teacher questionnaires. These 
will be factor analyzed and mean factor scores of certified and non-certified teachers as 
well as the corresponding student groups will be compared. 
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Analysis of Professional Activities and Involvement 

The telephone interviews yielded 12 categories of information related to the 
amount and type of teachers’ professional activities. Scoring rubrics for these categories 
have been developed and are being applied to the data. 

We will investigate patterns of professional activity of certified and non-certified 
teachers via a discriminant function analysis. We will not conduct any formal statistical 
analyses of the telephone interview! data related to how agencies make use of the expertise 
of NBCTs. The data will be catalogued and summarized so as to enable school systems to 
capitalize on the experiences of other systems. 
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